Current researches on the assessment of image aesthetic quality are based on visual content of images to give assessment results, ignoring the fact that aesthetics is a person's cognitive activity and not considering the user's understanding towards image semantic information during the evaluating process. In order to solve this problem, an approach to image aesthetic quality assessment based on semantic perception was proposed to apply both the object category information and scene category information of images to the aesthetic quality assessment. Using the transfer learning concept, a hybrid network integrating multiple features of the images was constructed. For each input image, the object category features, scene category features, and aesthetic features were extracted respectively by network, and the three features were combined to achieve better image aesthetic quality evaluation. The classification accuracy of the method on the AVA data set reached 89.5%, which was 19.9% higher than that of the traditional method, and the generalization performance on the CUKHPQ data set was greatly improved. The experimental results show that the proposed approach can achieve better classification performance on the aesthetic quality evaluation of images.